Directed interpolation and data post-processing

ABSTRACT

An encoding device evaluates a plurality of processing and/or post-processing algorithms and/or methods to be applied to a video stream, and signals a selected method, algorithm, class or category of methods/algorithms either in an encoded bitstream or as side information related to the encoded bitstream. A decoding device or post-processor utilizes the signaled algorithm or selects an algorithm/method based on the signaled method or algorithm. The selection is based, for example, on availability of the algorithm/method at the decoder/post-processor and/or cost of implementation. The video stream may comprise, for example, downsampled multiplexed stereoscopic images and the selected algorithm may include any of upconversion and/or error correction techniques that contribute to a restoration of the downsampled images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.18/175,122, filed on Feb. 27, 2023, which is a continuation of U.S.patent application Ser. No. 17/943,808, filed on Sep. 13, 2022 (now U.S.Pat. No. 11,677,985), which is a continuation of U.S. patent applicationSer. No. 16/831,647, filed on Mar. 26, 2020 (now U.S. Pat. No.11,477,480), which is a continuation of U.S. patent application Ser. No.16/261,196, filed on Jan. 29, 2019 (now U.S. Pat. No. 10,609,413), whichis a continuation of U.S. patent application Ser. No. 15/678,982, filedon Aug. 16, 2017, (now U.S. Pat. No. 10,194,172), which is acontinuation of U.S. patent application Ser. No. 14/954,891, filed onNov. 30, 2015 (now abandoned), which is a continuation of U.S. patentapplication Ser. No. 13/255,308, filed on Sep. 8, 2011, (now U.S. Pat.No. 9,729,899), which is the U.S. National Stage Entry for InternationalPatent Application No. PCT/US2010/031762, filed on Apr. 20, 2010, whichclaims the benefit of priority to U.S. Provisional Application No.61/170,995, filed Apr. 20, 2009, all of which are hereby incorporated byreference in their entirety.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION Field of Invention

The present invention relates to interpolation and post processing.

Discussion of Background

In many environments and applications, resolution of a signal may haveto be compromised. This could be done, for example, to reduce thebandwidth required to represent the signal, reduce decoding complexity,fit the characteristics of a certain display device etc. The resolutionof the signal could also be specified in a variety ofdomains/dimensions. In the case of a video signal, such could include,spatial (i.e., horizontal and vertical) resolution, temporal resolution,bit-depth, and even stereo information among others. In many cases,resolution in one dimension can be compromised in favor of another, e.g.in the case of interlace video signals, spatial resolution is sacrificedin favor of temporal resolution. For 3D applications, one compromise isto downsample spatially, using a variety of arrangements, and thenmultiplex the available stereo views. Downsampling can be horizontal,vertical, or even quincunx based, while the data can be packed orinterleaved together, i.e. using checkerboard multiplexing.

SUMMARY OF THE INVENTION

The present inventors have realized that current methods forreconstruction of downscaled or otherwise compromised video and otherdata can be severely limited in performance, or constrained in terms ofcomplexity, given that the conversion process has little or noinformation about the original nature and characteristics of thecontent.

In various embodiments, the present invention provides for thereconstruction of a downscaled or otherwise compromised video or otherdata back to it's original, or even a higher/different resolution orformat. For example, a 720p (1280×720) resolution video may be (or mayneed to be) upconverted to a 1080p (1920×1080) resolution for display ona higher resolution display. Similarly, a 3D signal comprising of twoquincunx sampled views multiplexed in a checkerboard arrangement or a 3Dsignal comprising of two horizontally or quincunx sampled views packedin a side by side arrangements may have to be upconverted to two fullresolution 3D views or a different sampling arrangement such as line byline.

FIG. 9 is an illustration of an exemplary quincunx sampling technique,where, for example, a full resolution image 905 (e.g., a left view of astereographic image pair) is operated on by a filter/function/conversionƒ(x) 920 and then quincunx sampled by sampler 930. The result is asample pattern that is, for example, as illustrated at 940. The samplepattern (of pixel data) may then be multiplexed with another quincunxsampled image (e.g., right view of the stereoscopic pair) or possiblyanother view or even an unrelated image, and then encoded. The processcan also be applied to images using a different sampling and packingmethod, such as horizontal sampling with side by side packing, orvertical sampling with line by line or over under packing.

The conversion process can be done using a variety of algorithms,including any one or more of, for example, separable or non-separablefilters, edge adaptive interpolation methods etc, which, in variousembodiments, may also require the analysis and characterization of thesignal.

FIG. 10 is an illustration of an interpolator for quincunx sampled dataaccording to an embodiment of the present invention. The process issimilar to other sampling and packing methods. As illustrated in FIG. 10, a bitstream 1005 is provided to a decoder 1050 that decodes a sampledimage 1080 (quincunx sampled image in this example), which is thenapplied to each of a series of filter operations 1085. The filteroperations 1085 may include any temporal, spatial or other upconversionprocess or any other processes described herein for any type ofup-conversion, error correction, color correction, or generalimprovement of the sampled image. The filter operation to be utilizedfor any particular pixel or region is, for example, specified bymetadata extracted by metadata extractor 1070 from the bitstream 1005.The result is, for example, a full resolution version 1090 of the image.

In the case of stereographic image pairs to be projected, the directedfilter operations may include color correction such that the colors incorresponding portions of the left and right images are consistent withthe director's intent. In such an embodiment/application, the colorcorrection is particularly important in projection systems usingdifferent color light sources or filters so that both the left and rightimage colors match as intended.

The present invention includes methods which enable improved conversionperformance and/or control conversion complexity by signaling to adevice, e.g. a decoder, a display, or other system, which conversionmethod, or methods, should be used. The methods may also be used toalter or control the behavior of the decoder in certain scenarios orenvironments such as in the case of controlling decoder complexity byaltering or even disabling certain tools that were originally mandatedby the decoding process, or, in the case of error concealment, where theinformation provided can now consider a more “informed” process forcorrecting errors in the signal. The signaling may be, for example, atthe sequence, group of pictures, picture, or region level. A group ofpictures can be of a fixed size or can be arbitrary. Similarly a regioncan be of fixed shape and size down to even a pixel, or could also bearbitrary. The invention is not restricted in only the methods forsignaling said parameters but also on how to derive the “best”, given apredefined criterion, parameters.

The present invention may be embodied as a method comprising the step ofreceiving a signal indicative of a processing technique to be utilizedon a data stream.

The processing technique may comprise, for example any one or more of aresolution conversion, controlling a complexity of processing, artifactreduction, error correction, error concealment, scaling, interpolation,an alteration of existing processing tools, and an enabling or disablingof at least one tool. The resolution conversion may comprise at leastone of spatial resolution conversion and temporal resolution conversion.Scaling may comprise at least one of de-interlacing, temporalinterpolation, and spatial interpolation. Error concealment may comprisea prediction of motion of a region in error. The signal may be furtherindicative of at least one of a spatial and temporal extent to which theprocessing technique is to be applied.

In one embodiment, the data stream is a video data stream and the extentcomprise whether the processing technique is applied to at least one ofall of the video, portions of the video, pictures of the video, andregions within the video. The pictures of the video may comprise, forexample, any of a frame of the video and a field of the video.

In one embodiment, the data stream comprises a video comprising at leasttwo different views and the signal further indicates that the processingtechnique is performed for at least one region within at least one frameof the video. The different views may comprise, for example, at leastone of stereographic views, two different images, a 2D image and depthinformation, multiple views of a 2D scene having differentcharacteristics such as resolution, bitdepth, or color information, andmultiple views of a 3D scene. The at least two different views may alsobe compressed and multiplexed within the data stream in a standardizedmotion picture format capable of single view video streams. Compressionof the views may comprise at least one of a sampling, filtering, anddecimation of the views. The compression of the views may also compriseat least one of horizontal, vertical filtering, and quincunx sampling.The compression of the views may also comprise both filtering andsampling. And sampling may, for example, comprise at least one ofhorizontal, vertical, quincunx, formula based, pattern based, andarbitrary sampling. Multiplexing may be done, for example, in at leastone of a checkerboard format, a quadrant based format, a column format,a row format, a side-by-side format, an over-under format, a formatbased on a pattern, and an alternative format.

In one embodiment, the signal is configured to identify at least one ofat least part of a decoding process and at least part of a post-decodingprocess to be applied to a region of an image or video scene havingcharacteristics on which the identified process operates efficiently.The characteristics may comprise at least one of texture, contour, edge,contrast, dynamic range, and brightness of the region.

In one embodiment, the signal is received from at least one of the datastream and side information related to the data stream. In anotherembodiment, the signaling may comprise a processing technique to beapplied to at least one of a sequence, group of pictures (GOP), andregion of one of a fixed, arbitrary, or varying size. The method mayfurther comprise the step of selecting at least one processing techniquebased on the signaled processing technique and applying the selectedprocessing technique to at least one of the sequence, GOP, and region.The step of selecting may be performed, for example, by a decoderconfigured to decode the data stream by applying the selected processingtechnique to at least one of the sequence, group of pictures (GOP), andregion. The step of selecting may also comprise selecting a processingtechnique in a different category of processing techniques than thesignaled processing technique but nearest to the signaled processingtechnique.

In one embodiment, the processing technique comprises a series ofprocessing techniques. The series of processing techniques may comprise,for example, at least one of an interpolation process, filtering, and ade-blocking process.

In one embodiment, the signaling may comprise signaling a family ofprocessing techniques, and the method may further comprise the steps ofselecting at least one processing technique from the signaled family andapplying the selected technique(s) to at least one of a sequence, groupof pictures (GOP), and a region of one of a fixed, arbitrary, or varyingsize.

In one embodiment, the processing technique was selected via apre-processing step prior to receiving the data stream. Thepre-processing may comprise, for example, studio work performed prior tomastering the data stream for media storage or broadcast. Thepre-processing may comprise an iterative performance of a set ofpotential processing techniques on a like data stream and an embeddingof an identifier of a selected processing technique in at least one ofthe data stream and side information related to the data stream. Thepre-processing may also be performed on-the-fly during a pre-processorplayback of a video captured by the data stream prior to, orcontemporaneously with, encoding and then storage on a medium ortransmission through any number of channels for decoding and playback byan end user.

The selected processing technique may also require additionalinformation generated by performing additional analysis and/orprocessing during decoding and playback of the data stream at thedecoder. Such analysis and/or processing may occur, for example, duringan on-the-fly decoding and playback of the encoded video for viewing byan end user viewer. Such analysis and/or processing may comprise, forexample, a refinement of analysis and/or processing performed at theencoder.

The selected processing technique may comprise, for example, at leastone of a least cost processing technique, a highest performanceprocessing technique, and a combination or tradeoff between cost andperformance. Cost may comprise, for example, at least one of complexity,area, power, and price. Performance may comprise, for example, at leastone of quality and speed.

In one embodiment, additional analysis comprises, for example, anadditional “assist/second” post-processing may also happen on thedecoder device given the information provided.

In another embodiment, the data stream comprises an encoded video datastream and the signaled processing technique was selected to enhance oneof a decoding and a post-process of decoding for at least one of apicture, Group Of Pictures (GOP), and region of the video data stream.The processing technique was selected, for example, at least one ofbefore, during, or after an encoding of the video data stream. The datastream may comprise, for example, a trans-coded encoded video stream,and the processing technique may have been selected at least one ofbefore, during, or after an encoding of the video data stream. Theprocessing technique may comprise, for example, parameters to be used inconjunction with the processing technique on a region of the video datastream, and the method may further comprise receiving a change inparameters for at least one of sub-regions of the region and a differentregion.

The invention may also be embodied as a method, comprising the steps of,selecting a mechanism for improving at least a portion of a video to bedecoded from an encoded video wherein the selected mechanism isconfigured to direct at least one of a decoding of the encoded video anda post-process applied to the video after decoding, and packaging atleast one of the selected mechanism and an identifier of the selectedmechanism as a directive signal into at least one of the encoded videoand side information of the encoded video. The selected mechanism maycomprise at least one of a process and parameters of a process. Theportion of the video may comprise at least one of a region andsub-region of the video.

In one embodiment, the portion of the video comprises at least one of animage of the video, a sequence of images of the video, a region of animage of the video, a dynamic region across frames of the video, and anysub-region(s) of the video. Alternatively, the portion of the videocomprises a region or sub-region of the video comprising a block. Inanother alternative, the portion of the video comprises a region orsub-region of the video comprising multiple blocks. The blocks may be,for example, non-contiguous. The blocks may also comprise at least partof a checkerboard pattern.

In one embodiment, the portion of the video comprises a geometricarrangement of video data samples across multiple frames of the video.The geometric arrangement may vary in at least one of size and shapebetween frames of the video.

In one embodiment, the portion of the video comprises an irregulararrangement of video data samples. In another embodiment, the portion ofthe video comprises co-located samples of a stereoscopic scene of thevideo. In yet another embodiment, the portion of the video comprises acuboid comprising M×K×N where M is width, K is height, and N is a numberof temporal samples. In yet another embodiment, the portion of the videocomprises one of a segment and an object tracked across multiple frames.In still yet another embodiment, the portion of the video is determinedby at least one of user input, segmentation, object tracking, scene cutdetection, edge detection, watershed, haursdorff method, K-means, motiondetection, motion estimation, motion analysis, and quality evaluation.In still yet another embodiment, the method may further comprise thestep of repeating the steps of selecting and packaging for a 2^(nd)portion of the video. The portion of the video may comprise a regularlyshaped region and the 2^(nd) portion of the video may comprise anirregularly shaped region. In one embodiment, the portion of the videoand the 2 nd portion of the video overlap. The selected mechanism maycomprise an instruction of how to average processing performed by themechanisms in the video overlap. The selected mechanism may alsocomprise an instruction of how to sequentially initiate the mechanismsin the video overlap.

In one alternative, the portions of the video are predefined and thepackaging contains no information identifying the portions. In anotheralternative, the portions of the video are defined using at least one ofa cuboid description, a vector based description, and a raster baseddescription. The portion of the video may be, for example, identifiedwith data encoded using an image format.

In another embodiment, the portion of the video is identified with dataencoded using a format comprising at least one of CGM, SVG, EPS, WMF,TIFF, BMP, JPEG, JPEG-2000, MPEG-2, and H.264, and MPEG4-AVC. In anotherembodiment, the portion of the video comprises a region identified in aprocess map. In another embodiment, the portion of the video comprises aregion mapped in at least one of an image or graphics format. In anotherembodiment, the invention includes packaging sequence instructions alongwith the directive signal. In another embodiment, the directive signalcomprises sequencing that indicates a preferred order of variousprocesses to be performed. In another embodiment, the directive signalmay comprise a sequencing of spatial up-sampling, temporal up-sampling,and de-blocking to be performed at least in part by the selectedmechanism. In another embodiment, the portion of the video comprises aregion divided into a series of sub-regions wherein the selectedmechanism comprises a Pt parameter corresponding to a first of thesub-regions and a 2^(nd) parameter corresponding to a second of thesub-regions. In another embodiment, the mechanism is selected based atleast in part on a complexity of the mechanism. The selected mechanismmay have, for example, a lowest complexity for a given qualitythreshold.

The present invention may also be embodied as a post-processing device,comprising a video input configured to receive a decoded video stream,an instruction mechanism configured to one of accept a directiveinstruction from a side information channel of the video stream andidentify a directive instruction from the decoded video stream, apost-processor configured to post-process the decoded video stream basedon the directive instruction, and a video output configured to outputthe post-processed video stream. The post-processing device may beintegrated, for example, into at least one of a display, a set-top box,and a media player. The directive instruction may comprise, for example,an identification of at least one region in the video and at least oneprocessing method to be applied to the region. The region may comprise,for example, sub-regions and the directive instruction may comprisedifferent parameters to be applied to the processing method forprocessing each sub-region. The directive signal may also identifyregions of different texture qualities and processing methodsspecifically for each texture quality.

In one embodiment, the directive signal comprises a filteridentification selected after testing a plurality of filters on thedecoded video stream. The filter identification may comprise anidentification of an entire category of filters. The post-processor mayalso be configured to select a filter from the category of filters andutilize the selected filter to post-process the decoded video stream.The post-processor is further configured to select a filter havingproperties closest to properties of the category of filters and utilizethe selected filter as at least part of the post-process.

In one embodiment, the post-processor is further configured to select afilter known to have properties that are at least one of close to orbetter than the filter identified by the filter identification. Inanother embodiment, the filter identification is selected based on atleast one of distortion and complexity. The distortion may be evaluatedin a number of ways, including via at least one of SAD, SSE, subjectiveevaluation, blockiness, variance, frequency characteristics,image/texture sharpness, edge quality, and artifacts either spatially ortemporally.

In one embodiment, testing is performed in an open-loop type testingdevice, and in another embodiment, testing is performed in a closed-looptype testing device. The testing may be performed, for example, for aplurality of regions of the video. The plurality of regions maycomprise, for example, temporal regions of the video. The plurality oftemporal regions comprises regions of changing shape and size.

In one embodiment, the regions of the video are determined by at leastone of segmentation, object tracking, scene cut detection, edgedetection, watershed, haursdorff method, K-means, motion detection,motion estimation, and motion analysis. The regions of the video may beidentified using at least one of CGM, SVG, EPS, WMF, TIFF, BMP, JPEG,JPEG-2000, MPEG-2, and H.264, and MPEG4-AVC.

The present invention may also be embodied as a system, comprising, anencoder configured to select at least one of a processing technique, acategory of processing techniques, and a series of processing techniquesto be utilized in a decoder and encode the selected techniques or anidentifier of the selected technique(s) in at least one of an encodedvideo stream and side information related to the encoded video stream,and a decoder configured to receive the encoded video stream and decodethe video stream using at least one of the selected technique(s). Theselected technique(s) are solely identified from the encoded videostream. The selected technique(s) may be identified, for example, by thedecoder from at least one of the related side information and theencoded video stream. The techniques may be selected based on at leastone of quality, complexity, and cost. The decoder may utilize theselected processing technique to select another processing technique.

In one embodiment, the decoder maintains an inventory of processingtechniques and selects a processing technique or technique(s) from theinventory most closely matching the technique(s) selected by theencoder. In one embodiment, the decoder selected technique is selectedbased at least on one of complexity, cost, and quality. The decoderselected technique may also be selected based at least in part oncomplexity in light of an amount of processing power available in thedecoder.

Portions of both the device and method may be conveniently implementedin programming on a general purpose computer, or networked computers,and the results may be displayed on an output device connected to any ofthe general purpose, networked computers, or transmitted to a remotedevice for output or display. In addition, any components of the presentinvention represented in a computer program, data sequences, and/orcontrol signals may be embodied as an electronic signal broadcast (ortransmitted) at any frequency in any medium including, but not limitedto, wireless broadcasts, and transmissions over copper wire(s), fiberoptic cable(s), and co-ax cable(s), etc.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1A is an illustration of fixed size temporal regions for metadatasignaling according to an embodiment of the present invention;

FIG. 1B is an illustration of adaptively sized temporal regions formetadata signaling according to an embodiment of the present invention;

FIG. 2A is an illustration of equally sized regions according to anembodiment of the present invention;

FIG. 2B is an illustration of adaptively defined regions according to anembodiment of the present invention;

FIG. 2C is an illustration of sub-regions within various regionsaccording to an embodiment of the present invention;

FIG. 3 is a diagram of an open loop system for embedding metadataaccording to an embodiment of the present invention;

FIG. 4 is a diagram of a closed loop system for embedding metadataaccording to an embodiment of the present invention;

FIG. 5 is a diagram of a filter selection process based on subjective orobjective distortion estimation according to an embodiment of thepresent invention;

FIG. 6 is a diagram of metadata based post-processor according to anembodiment of the present invention;

FIG. 7 is a diagram of a metadata based adaptive decoding that may beused for, for example, error concealment or decoder simplification,according to an embodiment of the present invention;

FIG. 8 is a diagram of a decoder according to an embodiment of thepresent invention;

FIG. 9 is an illustration of a quincunx sampling technique;

FIG. 10 is an illustration of an interpolator for quincunx sampled dataaccording to an embodiment of the present invention; and

FIG. 11 is a flow chart illustrating an evaluation and selection of animage region according to an embodiment of the present invention;

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present inventors have observed that the performance and behavior ofan algorithm may be quite different depending on the characteristics ofthe region of an image, video scene, or other data that the algorithm isapplied. In particular, in the case of a video, for a region R₀ with lowtexture characteristics algorithm A₀ may behave better and result inbetter image quality for, e.g., scaling than algorithm A₁, while A₁ maybehave better and result in better image quality for a different regionR₁. Although a decoder may try to determine which algorithm may performbetter given the characteristics of the region or even given thecharacteristics of the converted regions given algorithms A₀ and A₁,this process can be too complex and too time consuming to perform.Furthermore, given the fact that the decoder only has limitedinformation about the image/video, performance is always bounded by theavailable information. This can considerably limit the performance ofsuch a scheme.

In various embodiments, the present invention assists the decoder orpost-processor by providing additional information which can dictatewhich algorithm, algorithms, or algorithm groups/classes it should useto process each different region. This is done by analyzing andevaluating each algorithm and its performance or/and characteristics atbefore (e.g. using a pre-processor), during or after the encodingprocess, rating the behavior of each algorithm given the application,and then selecting and signaling the appropriate algorithm, and anyrelated parameters, for each region.

Signaling can be done within the bitstream (or data stream) itself(e.g., including signals or metadata in the bitstream) or could besignaled as side information (e.g. as a metadata file or stream). At thedecoder, the recommended processing information about each region can bedecoded and the appropriate algorithm or algorithms selected, containedin, or identified in the signals or metadata is used for performing thenecessary operation or operations that may be predetermined or alsoidentified in the signals or metadata.

Note that in the present invention, the decoder can be configured tomaintain control over the algorithms utilized. For example, a decoder orpost-processor may not always be obligated to use the selected oridentified algorithm or class of algorithms, and may choose to disregardor select similar algorithms depending on circumstances (such as, forexample, availability of certain algorithms and/or cost to implement).Furthermore, the decoder or post-processor may select to partially“ignore” the recommended algorithm, e.g. A₀, and instead use a differentalgorithm, e.g. A₁. A₁ may have similar properties with A₀, but might beless complicated, or may be known to have better performance. It is alsopossible that maybe A₀ is not available or not known, in which case thedecoder or post-processor may wish to use a default algorithm, e.g.A_(D), which is predefined by the system.

Region Signaling

At the pre-processor and/or the encoder stage each algorithm or eachalgorithm class is preferably evaluated before signaling. Evaluation canbe done at a region level, where a region can be any portion of thebitstream. For example, in a video bitstream, the region may be theentire sequence or image, a sub-sequence/video scene, a stereo framepair or a single frame/view, including sub-sampled views, or sub-regionsin an image. Sub-regions in an image could have a regular (e.g.triangular, rectangular etc) or irregular/random shape, can be of anysize (e.g. 8×8, 32×32, 4×64 etc), and can even extend temporally (e.g.an object tracked in time) or in space (e.g. an object in 3D space).

Regions could also be considered using sub-sampling, especially in thecase of sub-sampled and multiplexed data as is the scenario of quincunxsampled and checkerboard multiplexed video data for 3D stereo delivery.Regions could also be predefined, e.g. by a user. For example, an entiresequence can be identified as a single region, or a region could specifya collection of N subsequent frames such as in the case of a fixed Groupof Pictures (GOP) of size N.

Regions may comprise, for example, a spatial regional segmentation usedfor processing to determine signaling for that segment. As shown in FIG.2A, M×K rectangular regions of equal size (width/M)×(height/K) wherewidth is the width and height is the height of the frame respectivelycan also be used. A user may even specify that all individual pixels, oreven all individual samples (e.g. separating luma and chroma samples)that correspond to a region.

Other combinations may also be utilized, including, for example,grouping together co-located samples in a stereo pair, a cuboid of sizeM×K×N where now N corresponds to temporal samples, etc. Regions may alsobe adaptively selected and defined and basically be of an arbitraryshape. As shown in FIG. 2B, adaptively defined regions A₀-A₄ areillustrated. In various embodiments, regions can be selected using oneor more of a variety of algorithms such as scene cut detection,image/video segmentation, object tracking, etc.

Algorithms for segmentation may include techniques using edge detection,watershed methods, the haursdorff distance, K-means, and could alsoconsider motion detection/estimation and analysis. Motion estimation maybe done using a variety of algorithms and models including affine,parabolic, translational among others, and could be block or regionbased, done in the spatial or frequency domain (e.g. using phasecorrelation) etc.

The present invention does not restrict the application of otheralgorithms for the determination of such regions. Regions may also beclassified given their illumination and chroma characteristics, texture,or be based on semantic information and user interaction. A combinationof arbitrary shaped and regular shaped regions could also be used. Forexample, a region could be of fixed temporal length N but its spatialshape may change for every frame.

As illustrated in FIG. 2C, a region may also be further separated, insub-regions. In this scenario, a system may specify first a set ofprimary regions (e.g., similar to the concept of a Macroblock in a videocoding system), to which a particular processing method needs to beapplied, while the same region can be, optionally, further separatedinto smaller sub-regions for which different parameters for theprocessing method can be used (e.g. the same algorithm but with adifferent strength filter or different thresholds). Regions may alsooverlap, in which case multiple algorithms can be applied to overlappingareas which can then be combined. Combining could be done afterprocessing using simple averaging, or could be based on a weightedaveraging method that could be based on the distance of the pixel fromthe center of each region, the characteristics and expected performanceof the algorithm and the characteristics of the overlapping pixels orregions. In one embodiment, different algorithms are applied in aspecified order to each region including the overlapped area.

If the regions are predefined, there is no need to encode anyinformation about their characteristics (e.g. shape, size, overlap etc).However, if these are not predefined, then it is required that suchinformation is encoded and is available to the decoder. Encoding is notnecessarily complex, especially for regularly shaped regions and caninclude information such as, for the case of a cuboid number ofhorizontal (M), vertical (K), and temporal (N) segments, or be morecomplicated such as using a vector or raster based description of theregions. For example, all regions could be described using the computergraphics metafile file format (CGM), the Scalable Vector Graphics (SVG)format, Encapsulated Postscript (EPS) or Windows Metafile (WMF), orother vector graphics formats. Similarly, for a raster baseddescription, we could use TIFF or BMP, JPEG, JPEG-2000, or even videocodec representations using MPEG-2 or H.264/MPEG-4 AVC. In thesescenarios prediction, both spatial and temporal, for representing theregions can be used. Both lossless and lossy representations could beconsidered where in the lossy case a different algorithm mayinadvertently be used. However, if this process is done carefully and isknown in the encoder, then this would not result in any issuesespecially if any processing is performed iteratively. In both vectorand raster representations, the representation can be kept in a highresolution or could be subsampled down to 1×1 sample.

In various embodiments an image can be characterized and represented bymultiple region representations/maps, which may or may not be related,with each representation relating to a different post/interpolationprocess that needs to be applied onto the image. For example, one regionrepresentation can specify spatial up-sampling, a second regionrepresentation can specify temporal up-sampling/frame rate conversion,while a third one can specify de-blocking or de-noising. The sequence ofwhich process should be performed first may depend on the order of therepresentations as they are found in the bitstream, or could bedescribed within the bitstream itself with additional syntax. Forexample, in the above scenario we can signal to perform first thespatial up-sampling followed by temporal up-sampling and thende-blocking, or we can signal that de-blocking should be performedfirst, followed by spatial up-sampling, and then temporal up-sampling.

Analysis and Filter Selection

The present invention may be configured to signal a variety ofpost-processing methods including the preferred/recommended spatial ortemporal up-scaling or downscaling method and its parameters,de-blocking, and de-noising among others, or even suggested(non-normative) simplifications in the decoding process and recommendedmethods for performing error concealment. The method and parameters tobe signaled for a region can be selected in a variety of ways. Inparticular, the parameters could be fixed (e.g. the same method M₀ forthe entire sequence or for regions A₀ in all frames, while method M₁ isused for regions A₁), changed periodically (e.g. every N frames methodM₀ in the above example is switched to method M_(0,N)), or could beadaptive given certain properties of the signal or given user input. Inone embodiment, the method is selected given the texture, edge,illumination, color, and motion characteristics of a scene. As anexample, in a high textured region a “softer” de-blocking method andparameters can be signaled, while in a low textured region a different,more aggressive de-blocker and parameters can be signaled instead.Similarly, in a region characterized by vertical edges an interpolationmethod that is more biased towards the vertical direction could besignaled, while in a region with diagonal edges, an interpolation methodbiased diagonally can be used instead.

In a different example, for a scene with low motion, a motioncompensated method could be signaled for temporal interpolation, whilefor high motion, frame replication or frame blending can be used. Thisdecision requires an analysis process to be performed prior to thedecision. The analysis can be performed with a variety of methods, suchas texture analysis (e.g. variance based or frequency analysis), using amotion estimation method including methods such as affine, parabolic,and translational (e.g. block based) motion estimation, and illuminationchange characterization among others.

In an alternative embodiment, the decision can be performed using adistortion and/or complexity analysis of each method. The method thatresults in the lowest distortion and/or complexity can be selected asthe method to be signaled. In particular, in an up-conversion examplesuch as 720p to 1080p, or horizontally sampled and Side by Sidemultiplexed or quincunx sampled and checkerboard multiplexed 3D videodata to a full resolution stereo pair, all available up-sampling filtersare tested for every region. The filter that results in an up-convertedregion that resembles as closely as possible to the originalup-converted signal and, optionally, requires the least or a desiredcomplexity for the conversion, is selected for signaling.

The structure of a distortion/complexity based model is illustrated inFIG. 5 , where an input source 505 is processed through a formatconversion 510, an encoder 540, and then a decoder 550. The decodedstream is then divided into regions and each region is forwarded to aset of filters 565 (e.g., filter 1 . . . N)(Region Selection 560). TheRegion Selection 560 may contain one or more algorithms for regionallydividing the decoded stream including predetermined regions and/oradaptively determined regions including any based on regionalbrightness, contrast, edges, motion, and/or others. After filtering adistortion and/or complexity analysis is performed 570. The distortionanalysis may be the same or tailored for each filter type. The filterselection mechanism 530 then evaluates the results of thedistortion/complexity analysis for each region. The best match based ona distortion/complexity selection criterion is then provided to theencoder 540 for production of the output bitstream 575. The selectedbest match is illustrated as being placed and/or encoded into the outputbitstream 575, but may be provided in other mechanisms (not shown),including side information and/or metadata related to the outputbitstream 575.

The selection can be based on distortion, e.g. compared to the original,but also jointly on rate, distortion, and even complexity of the method.The rate can include the bits that may be required to signal the methodand its parameters, assuming that the encoding of the method mayconsider compression (e.g. prediction and entropy coding). Rate may alsoaccount for the bits of the actual video signal since it might bepossible to consider multiple alternative representations of the samesignal which may require different number of bits.

Different methods, as specified by this invention, may be appropriatefor the different representations. In this scenario an encoder mayselect both the best representation (given a joint rate, distortion, andcomplexity criterion), but also the best post-processing method jointly.Distortion may be based on one or more of a variety of metrics, such asthe Sum of Absolute Differences (SAD), Sum of Square Errors (SSE), orother metrics including subjective evaluation. Distortion could also bebased on measuring a variety of characteristics of the image afterpost-processing, including blockiness, variance, frequencycharacteristics, image/texture sharpness, edge quality, or through thedetection and measurement of artifacts that may have been created due tothe usage of a particular method etc. Such measurements can be veryuseful in scenarios where the original content might not be available oris available but only at a different, i.e. lower, resolution or format.As an example, consider an up-conversion of a 30p signal to a 60p signal(increase frame rate by 2×). In this scenario, detection of temporalartifacts can be very useful to determine the best method to signal forevery frame or region. In one embodiment, the present invention utilizesat least one of a quantity and a quality of detected temporal artifactscaused by various methods to select which of the methods or classes ofmethods are signaled.

Selecting the best method could be done using a full search approach,i.e. test all available processing methods compared to the original orgiven the available evaluation metrics applicable to this process.However, fast methods could also be considered which can considerablyreduce complexity. In particular, instead of testing all possiblemethods for processing a region, an analysis of a region and given itscharacteristics may be used to provide guidance on a selection ofmethods that are appropriate for testing for that region. For example,for frame rate conversion, the invention may first perform a simplemotion estimation process, and given the motion information determine ifit is needed or necessary to test any temporal (motion compensated),motion adaptive, or only basic methods such as frame blending or framereplication. Motion estimation could consist of a block based schemesuch as the Enhanced Predictive Zonal Search (EPZS) algorithm, or couldjust rely on a simple frame/region difference.

FIG. 11 is a flow chart illustrating an evaluation and selection of animage region according to an embodiment of the present invention. Atstep 1110, an image or video is segmented into regions. The regions arepredetermined or determined based on any of the techniques describedherein. A loop 1120 is set up to iteratively examine each region (1120),and parameters are initialized (step 1130). For each of a plurality ofalgorithms, methods, or processes, a second loop is set up (1140). Foreach region, each algorithm/method/process is performed to determine thebest mode or result(s). In this example, a distance function (dist) iscomputed (step 1150) and compared to previously computed dist functions(step 1160). The best algorithm for each region may be saved and thenprovided for encoding or a best algorithm across all regions may beprovided for encoding (e.g., steps 1170 and 1195).

The characteristics of neighboring regions, including the methodsselected for these regions, if already known, could also be consideredto speed up the process. If, for example, the regions on the left andabove have selected method A0 then it is very likely that also thismethod might be appropriate for the current region. Therefore, invarious embodiments, the invention includes prioritization of likelymethods when testing, or the consideration and update of a distortionthreshold, which can be based on the neighboring regions and can be usedto test if this mode or method should also be considered for the currentregion. Alternatively, a pattern of shifting parameters in known regionsmay also be recognized and used to predict a slightly different set ofparameters for a current region to be tested.

The process may also prioritize first all other similar methods to A0.Similar considerations can be made if the neighboring regions have useddifferent processing methods. In this scenario, all methods used in theneighboring regions can be considered as being more probable than otheravailable methods. Probability can change given also the characteristicsof the region and the relationship of the characteristics of this regionwith its neighbors. For example, if the region is highly textured, thenit is likely that it will have more correlation with neighbors, temporalor spatial, with also similar texture. The higher the similarity withthe neighbors (motion, texture, color etc), the higher the probabilitythat also the processing method that should be applied should be thesame or at least similar.

In one embodiment, the method used is applied after decoding of acompressed signal, selection can be based on an open loop architecture.For example, in FIG. 3 , given only the original signal and the effecteach method would have on that signal. As illustrated in the FIG. 3example, an Input Source 305 undergoes a format conversion 310. Suchconversion may be for example, a horizontal or a quincunx sampling andside by side or checkerboard packing of a pair of stereographic images.The conversion is or may be regionally or otherwise segmented. A regionis selected (e.g., region selection 320), and each region in turn (or inparallel) is applied to series of filters (or methods, algorithms, orprocesses) (e.g., filters 1 . . . N). The filter having one of thehighest quality, least cost, or predetermined combination of both orother factors is then selected and provided to the encoder 340 where theselection is provided as either a signal encoded (or placed) in theoutput bitstream 345 or provided as side information (not shown) relatedto the output bitstream 345. The signal may contain, for example, theselected filter, an identifier of the filter, or an identifier of aclass or category of filters that help convert the signal to itsoriginal (e.g. full resolution) or other format.

In another embodiment, selection is based on a closed loop architecture(see FIG. 4 ). In a closed loop architecture, selection is based on theperformance of the method given the actual decoded signal, whichcommonly results in improved performance compared to an open loopsystem. As illustrated in the example of FIG. 4 , an input source 405 isprocessed through a format conversion 410, an encoder 440, and then adecoder 450. The decoded stream is then divided into regions and eachregion is forwarded to a set of filters 465 (e.g., filter 1 . . .N)(Region Selection 460). The Region Selection 460, may contain one ormore algorithms for regionally dividing the decoded stream includingpredetermined regions and/or adaptively determined regions including anybased on regional brightness, contrast, edges, motion, and/or others.The filter selection mechanism then compares each region after decodingand filtering to a corresponding region in the original input. The bestmatch based on a selection criteria is then provided to the encoder 440for production of the output bitstream 455. The selected best match isillustrated as being placed and/or encoded into the output bitstream455, but again, may be provided in other mechanisms such as, forexample, metadata or side information (not shown) related to the outputbitstream 455.

The signaling can specify specific filters or methods. It can also,however specify a class/type of filters or methods which can providefurther flexibility to a decoder or post-processor. For example, insteadof signaling that a separable filter with specific coefficients is usedfor up-conversion of the image, the system can specify that anyseparable filter or a separable filter of a certain length and above orbelow would result in acceptable performance. The decoder orpost-processor is then free to select which filter to use, givencomplexity or availability.

If a filter is not available, then a decoder/post-processor may alsohave the option to select the method that is the most similar one to themethod signaled (e.g. for the separable filter example, a non-separablefilter of the same length), or use a previously defined “default” filterthat is specified in the system. The decoder/post-processor, can alsoselect the method or completely ignore the metadata information givenother characteristics of the process or the device. In particular, givenpower/battery status of the device, the system may select to use ahigher or a lower complexity algorithm.

In fact, the present invention may also be structured to also signal themetadata alternative modes that can be used given power, or complexityfootprints of a device. For example, the present invention may includeembedding a number of alternative maps in the signal (e.g., 2 or more).Each alternative map targeting a different device (e.g. one map for ahigh definition LCD TV, another for a Plasma, another for a PC, an iPodor iPhone etc), and battery status. The device can detect itscorresponding map, i.e. using a signature in the metadata that specifiesthe device, and perform its corresponding and optimized processing. Itis apparent that for the encoding system, all devices need to beconsidered during the optimization and method selection process.

FIG. 6 is a diagram of metadata based post-processor according to anembodiment of the present invention. In this example, an encodedbitstream 605 is decoded (decoder 650) and provided to a series offilters each of which process the decoded signal. A metadata extractor670 identifies a previously selected filter to be utilized for thecurrent portion of the decoded bitstream being filtered and selects theoutput of the filtering processes from the appropriate filter. In otherembodiments, the metadata extractor 670 may be replaced with, forexample, a side information reader which provides similar informationfrom a side information data stream related to the input bitstream 605.The appropriate filter is the filter identified by the metadata/sideinformation, or, for example, a filter closest to the characteristics ofthe previously selected filter. The selected filter output then providesthe output decoded and filtered bitstream 680.

FIG. 7 is a diagram of a metadata based adaptive decoding that may beused for, for example, error concealment or decoder simplification,according to an embodiment of the present invention. An input bitstream705 is provided to a series of decoding mechanisms 720 (e.g., decoders 1. . . N) and a metadata extractor 770. The metadata extractor 770extracts a signal identifying the previously selected or preferreddecoder, which is used to select an appropriate output from the seriesof decoding mechanisms 720. The previously selected decoder may be aspecific decoder or a type of decoder, and the decoder output ultimatelyselected for the output bitstream 785 may be the decoder previouslyselected, a decoder of a same class or type, an equivalent decoder, ageneric decoder, or a decoder optimized with parameters selected toalter performance of the decoder to more closely match the previouslyselected decoder, or others.

It may be desirable, for some applications, to retain or alter themetadata information after processing, re-encoding, and trans-coding ofthe image data. In particular, if the content is rescaled and thenre-encoded, the metadata could be retained as is, or adjusted given thescaling applied but also the characteristics of the encoding (e.g.quantization parameters and performance of motion estimation andcompensation) and artifacts introduced by this process. This can reducethe complexity of reevaluating the performance of the availableprocessing methods. The characteristics of the recommended methods forpost-processing in the original signal can also provide hints in thesecondary encoding on whether any additional but also which type ofpost-processing may be applied at a subsequent decoder.

FIG. 8 is a diagram of an exemplary decoder 800 according to anembodiment of the present invention. The decoder 800 utilizes metadata“hints.” The decoder 800 includes a Metadata extractor 810 that extractsmetadata that is utilized in a plurality of processes within thedecoder, such as, for example, a decoding process (e.g., an inversetransform operation 820), a loop filter 840, a disparity compensator(providing disparity compensation) 830, and/or a post-processor 850among others. The metadata provides one or more of a selection, data,and/or parameters for use in the decoder 800, loop filter 840, disparitycompensator 830, and/or post processor 850. The meta data drives theabove processes, for example, to reduce complexity if these areconsidered safe to do so (e.g., such that they would produce minimaldrift and/or quality impact). Metadata similarly encoded in thebitstream (or provided in side information) may be utilized for anyother process or sub-process within the decoder or other outsidefunctions.

In describing preferred embodiments of the present invention illustratedin the drawings, specific terminology is employed for the sake ofclarity. However, the present invention is not intended to be limited tothe specific terminology so selected, and it is to be understood thateach specific element includes all technical equivalents which operatein a similar manner. For example, when describing a decoder, any otherequivalent device, or device having a decoder therein, such as a mediaplayer (e.g., DVD, Blu-ray, iPod, computer, etc), tuner (e.g., PAL,ATSC, etc), set-top box, display or television, or other device havingan equivalent function or capability, whether or not listed herein, maybe substituted therewith. Furthermore, the inventors recognize thatnewly developed technologies not now known may also be substituted forthe described parts and still not depart from the scope of the presentinvention. All other described items, including, but not limited todevices and processors (e.g., post-processors), conversion methods,up-conversion methods, processing, algorithms, selection mechanisms,artifact identification, etc., should also be considered in light of anyand all available equivalents.

Portions of the present invention may be conveniently implemented usinga conventional general purpose or a specialized digital computer ormicroprocessor programmed according to the teachings of the presentdisclosure, as will be apparent to those skilled in the computer art.

Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art. The invention may also beimplemented by the preparation of application specific integratedcircuits or by interconnecting an appropriate network of conventionalcomponent circuits, as will be readily apparent to those skilled in theart based on the present disclosure.

The present invention includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to control, or cause, a computer to perform any of the processesof the present invention. The storage medium can include, but is notlimited to, any type of disk including floppy disks, mini disks (MD's),optical discs, DVD, HD-DVD, Blu-ray, CD-ROMS, CD or DVD RW+/−,micro-drive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs,DRAMs, VRAMs, flash memory devices (including flash cards, memorysticks), magnetic or optical cards, SIM cards, MEMS, nanosystems(including molecular memory ICs), RAID devices, remote datastorage/archive/warehousing, or any type of media or device suitable forstoring instructions and/or data.

Stored on any one of the computer readable medium (media), the presentinvention includes software for controlling both the hardware of thegeneral purpose/specialized computer or microprocessor, and for enablingthe computer or microprocessor to interact with a human user or othermechanism utilizing the results of the present invention. Such softwaremay include, but is not limited to, device drivers, operating systems,and user applications. Ultimately, such computer readable media furtherincludes software for performing the present invention, as describedabove.

Included in the programming (software) of the general/specializedcomputer or microprocessor are software modules for implementing theteachings of the present invention, including, but not limited to,analysis of methods, signaling of methods, classes of methods,categories of methods, preparation and transmission of signals,selection of signals at a decoder or post-processor, implementingalgorithms identified in signals that perform at least part of apost-process or decoding operation, and the display, storage, orcommunication of results according to the processes of the presentinvention (which may include, for example, delivery of post-processedvideo signals to a display).

Various embodiments of the present invention may relate to one or moreof the Enumerated Example Embodiments (EEEs) below, each of which areexamples, and, as with any other related discussion provided above,should not be construed as limiting any claim or claims provided yetfurther below as they stand now or as later amended, replaced, or added.

Likewise, these examples should not be considered as limiting withrespect to any claim or claims of any related patents and/or patentapplications (including any foreign or international counterpartapplications and/or patents, divisionals, continuations, re-issues,etc.). Examples:

Enumerated Example Embodiment 1 (EEE1). A method comprising the step ofreceiving a signal indicative of a processing technique to be utilizedon a data stream.

EEE2. The method according to EEE1, wherein the processing techniquecomprises at least one of resolution conversion, controlling acomplexity of processing, artifact reduction, error correction, errorconcealment, scaling, interpolation, an alteration of existingprocessing tools, and an enabling or disabling of at least one tool.

EEE3. The method according to EEE2, wherein the resolution conversioncomprises at least one of spatial resolution conversion and temporalresolution conversion.

EEE4. The method according to EEE2, wherein scaling comprises at leastone of de-interlacing, temporal interpolation, and spatialinterpolation.

EEE5. The method according to EEE2, wherein error concealment comprisesa prediction of motion of a region in error.

EEE6. The method according to EEE1, wherein the signal is furtherindicative of at least one of a spatial and temporal extent to which theprocessing technique is to be applied.

EEE7. The method according to EEE6, wherein the data stream is a videodata stream and the extent comprises whether the processing technique isapplied to at least one of all of the video, portions of the video,pictures of the video, and regions within the video.

EEE8. The method according to EEE7, wherein pictures of the videocomprise any of a frame of the video and a field of the video.

EEE9. The method according to EEE1, wherein the data stream comprises avideo comprising at least two different views and the signal furtherindicates that the processing technique is performed for at least oneregion within at least one frame of the video.

EEE10. The method according to EEE9, wherein the different viewscomprise at least one of stereographic views, two different images, a 2Dimage and depth information, multiple views of 2D scene, and multipleviews of a 3D scene.

EEE11. The method according to EEE9, wherein the at least two differentviews are compressed and multiplexed within the data stream in astandardized motion picture format capable of single view video streams.

EEE12. The method according to EEE9, wherein compression of the viewscomprises at least one of a sampling, filtering, and decimation of theviews.

EEE13. The method according to EEE9, wherein compression of the viewscomprises at least one of horizontal, vertical filtering, and quincunxsampling.

EEE14. The method according to EEE9, wherein compression of the viewscomprises both filtering and sampling.

EEE15. The method according to EEE14, wherein the sampling comprises atleast one of horizontal, vertical, quincunx, formula based, patternbased, and arbitrary sampling.

EEE16. The method according to EEE14, wherein the multiplexing is donein at least one of a checkerboard format, a quadrant based format, acolumn format, a row format, a side-by-side format, an over-underformat, a format based on a pattern, and an alternative format.

EEE17. The method according to EEE1, wherein the signal identifies atleast one of at least part of a decoding process and at least part of apost-decoding process to be applied to a region of an image or videoscene having characteristics on which the identified process operatesefficiently.

EEE18. The method according to EEE17, wherein the characteristicscomprise at least one of texture, contour, edge, contrast, dynamicrange, and brightness of the region.

EEE19. The method according to EEE1, wherein the signal is received fromat least one of the data stream and side information related to the datastream.

EEE20. The method according to EEE1, wherein the signaling comprises aprocessing technique to be applied to at least one of a sequence, groupof pictures (GOP), and region of one of a fixed, arbitrary, or varyingsize.

EEE21. The method according to EEE20, wherein the method furthercomprises the step of selecting at least one processing technique basedon the signaled processing technique and applying the selectedprocessing technique to at least one of the sequence, GOP, and region.

EEE22. The method according to EEE21, wherein the step of selecting isperformed by a decoder configured to decode the data stream by applyingthe selected processing technique to at least one of the sequence, groupof pictures (GOP), and region.

EEE23. The method according to EEE21, wherein the step of selectingcomprises selecting a processing technique in a different category ofprocessing techniques than the signaled processing technique but nearestthe signaled processing technique.

EEE24. The method according to EEE1, wherein the processing techniquecomprises a series of processing techniques.

EEE25. The method according to EEE24, wherein the series of processingtechniques comprises at least one of an interpolation process,filtering, and a de-blocking process.

EEE26. The method according to EEE1, wherein the signaling comprisessignaling a family of processing techniques, and the method furthercomprises the steps of selecting at least one processing technique fromthe signaled family and applying the selected technique(s) to at leastone of a sequence, group of pictures (GOP), and a region of one of afixed, arbitrary, or varying size.

EEE27. The method according to EEE1, wherein the processing techniquewas selected via a pre-processing prior to receiving the data stream.

EEE28. The method according to EEE27, wherein the pre-processingcomprises studio work performed prior to mastering the data stream formedia storage or broadcast.

EEE29. The method according to EEE27, wherein the processing techniqueis performed on-the-fly from a playback of the data stream.

EEE30. The method according to EEE27, wherein the pre-processingcomprises an iterative performance of a set of potential processingtechniques on a like data stream and an embedding of an identifier of aselected processing technique in at least one of the data stream andside information related to the data stream.

EEE31. The method according to EEE27, wherein the pre-processing isperformed on-the-fly during a playback of a video captured by the datastream.

EEE32. The method according to EEE27, wherein the selected processingtechnique comprises at least one of a least cost processing technique, ahighest performance processing technique, and a combination or tradeoffbetween cost and performance.

EEE33. The method according to EEE32, wherein cost comprises at leastone of complexity, area, power, and price.

EEE34. The method according to EEE32, wherein performance comprises atleast one of quality and speed.

EEE35. The method according to EEE1, wherein the data stream comprisesan encoded video data stream and the signaled processing technique wasselected to enhance one of a decoding and a post-process of decoding forat least one of a picture, Group Of Pictures (GOP), and region of thevideo data stream.

EEE36. The method according to EEE35, wherein the processing techniquewas selected at least one of before, during, or after an encoding of thevideo data stream.

EEE37. The method according to EEE35, wherein the data stream comprisesa trans-coded encoded video stream.

EEE38. The method according to EEE37, wherein the processing techniquewas selected at least one of before, during, or after a transcoding ofthe video data stream.

EEE39. The method according to EEE35, wherein the processing techniquecomprises parameters to be used in conjunction with the processingtechnique on a region of the video data stream, and the method furthercomprises receiving a change in parameters for at least one ofsub-regions of the region and a different region.

EEE40. A method, comprising the steps of:

-   -   selecting a mechanism for improving at least a portion of a        video to be decoded from an encoded video wherein the selected        mechanism is configured to direct at least one of a decoding of        the encoded video and a post-process applied to the video after        decoding; and    -   packaging at least one of the selected mechanism and an        identifier of the selected mechanism as a directive signal into        at least one of the encoded video and side information of the        encoded video.

EEE41. The method according to EEE40, wherein the selected mechanismcomprises at least one of a process and parameters of a process.

EEE42. The method according to EEE40, wherein the portion of the videocomprises at least one of a region and sub-region of the video.

EEE43. The method according to EEE40, wherein the portion of the videocomprises at least one of an image of the video, a sequence of images ofthe video, a region of an image of the video, a dynamic region acrossframes of the video, and any sub-region(s) of the video.

EEE44. The method according to EEE40, wherein the portion of the videocomprises a region or sub-region of the video comprising a block.

EEE45. The method according to EEE40, wherein the portion of the videocomprises a region or sub-region of the video comprising multipleblocks.

EEE46. The method according to EEE45, wherein the blocks arenon-contiguous.

EEE47. The method according to EEE45, wherein the blocks are comprisedat least part of a checkerboard pattern.

EEE48. The method according to EEE40, wherein the portion of the videocomprises a geometric arrangement of video data samples across multipleframes of the video.

EEE49. The method according to EEE48, wherein the geometric arrangementvaries in at least one of size and shape between frames of the video.

EEE50. The method according to EEE40, wherein the portion of the videocomprises an irregular arrangement of video data samples.

EEE51. The method according to EEE40, wherein the portion of the videocomprises co-located samples of a stereoscopic scene of the video.

EEE52. The method according to EEE40, wherein the portion of the videocomprises a cuboid comprising M×K×N where M is width, K is height, and Nis a number of temporal samples.

EEE53. The method according to EEE40, wherein the portion of the videocomprises one of a segment and an object tracked across multiple frames.

EEE54. The method according to EEE40, wherein the portion of the videois determined by at least one of user input, segmentation, objecttracking, scene cut detection, edge detection, watershed, haursdorffmethod, K-means, motion detection, motion estimation, motion analysis,and quality evaluation.

EEE55. The method according to EEE40, further comprising the step ofrepeating the steps of selecting and packaging for a 2^(nd) portion ofthe video.

EEE56. The method according to EEE55, wherein the portion of the videocomprises a regularly shaped region and the 2^(nd) portion of the videocomprises an irregularly shaped region.

EEE57. The method according to EEE55, wherein the portion of the videoand the 2^(nd) portion of the video overlap.

EEE58. The method according to EEE57, wherein the selected mechanismcomprises an instruction of how to average processing performed by themechanisms in the video overlap.

EEE59. The method according to EEE57, wherein the selected mechanismcomprises an instruction of how to sequentially initiate the mechanismsin the video overlap.

EEE60. The method according to EEE57, wherein the portions of the videoare predefined and the packaging contains no information identifying theportions.

EEE61. The method according to EEE57, wherein the portions of the videoare defined using at least one of a cuboid description, a vector baseddescription, and a raster based description.

EEE62. The method according to EEE40, wherein the portion of the videois identified with data encoded using an image format.

EEE63. The method according to EEE40, wherein the portion of the videois identified with data encoded using a format comprising at least oneof CGM, SVG, EPS, WMF, TIFF, BMP, JPEG, JPEG-2000, MPEG-2, and H.264,and MPEG4-AVC.

EEE64. The method according to EEE40, wherein the portion of the videocomprises a region identified in a process map.

EEE65. The method according to EEE40, wherein the portion of the videocomprises a region mapped in at least one of an image or graphicsformat.

EEE66. The method according to EEE40, further comprising the step ofpackaging sequence instructions along with the directive signal.

EEE67. The method according to EEE40, wherein the directive signalcomprises sequencing that indicates a preferred order of variousprocesses to be performed.

EEE68. The method according to EEE40, wherein the directive signalcomprises a sequencing of spatial up-sampling, temporal up-sampling, andde-blocking to be performed at least in part by the selected mechanism.

EEE69. The method according to EEE40, wherein the portion of the videocomprises a region divided into a series of sub-regions wherein theselected mechanism comprises a Pt parameter corresponding to a first ofthe sub-regions and a 2 nd parameter corresponding to a second of thesub-regions.

EEE70. The method according to EEE40, wherein the mechanism is selectedbased at least in part on a complexity of the mechanism.

EEE71. The method according to EEE40, wherein selected mechanism has alowest complexity for a given quality threshold.

EEE72. The method according to EEE40, wherein:

-   -   said method is embodied in a set of instructions stored on an        electronically readable media;    -   said instructions, when loaded into a processing device, cause        the processing device to perform the steps of said method.

EEE73. The method according to EEE72, wherein said instructions arecompiled computer instructions stored as an executable program on saidelectronically readable media.

EEE74. The method according to EEE40, wherein said method is embodied ina set of electronically readable instructions stored in an electronicsignal.

EEE75. A post-processing device, comprising:

-   -   a video input configured to receive a decoded video stream;    -   an instruction mechanism configured to one of accept a directive        instruction from a side information channel of the video stream        and identify a directive instruction from the decoded video        stream;    -   a post-processor configured to post-process the decoded video        stream based on the directive instruction; and    -   a video output configured to output the post-processed video        stream.

EEE76. The post-processing device according to EEE75, wherein thepost-processing device is integrated into at least one of a display, aset-top box, and a media player.

EEE77. The post-processing device according to EEE75, wherein thedirective instruction comprises an identification of at least one regionin the video and at least one processing method to be applied to theregion.

EEE78. The post-processing device according to EEE77, wherein the regioncomprises sub-regions and the directive instruction comprises differentparameters to be applied to the processing method for processing eachsub-region.

EEE79. The post-processing device according to EEE77, wherein thedirective signal identifies regions of different texture qualities andprocessing methods specifically for each texture quality.

EEE80. The post-processing device according to EEE75, wherein thedirective signal comprises a filter identification selected aftertesting a plurality of filters on the decoded video stream.

EEE81. The post-processing device according to EEE80, wherein the filteridentification comprises an identification of an entire category offilters.

EEE82. The post-processing device according to EEE81, wherein thepost-processor is further configured to select a filter from thecategory of filters and utilize the selected filter to post-process thedecoded video stream.

EEE83. The post-processing device according to EEE82, wherein thepost-processor is further configured to select a filter havingproperties closest to properties of the category of filters and utilizethe selected filter as at least part of the post-process.

EEE84. The post-processing device according to EEE80, wherein thepost-processor is further configured to select a filter known to haveproperties that are at least one of close to or better than the filteridentified by the filter identification.

EEE85. The post-processing device according to EEE80, wherein the filteridentification is selected based on at least one of distortion andcomplexity.

EEE86. The method according to EEE84, wherein distortion is evaluatedvia at least one of SAD, SSE, subjective evaluation, blockiness,variance, frequency characteristics, image/texture sharpness, edgequality, and artifacts either spatially or temporally.

EEE87. The post-processing device according to EEE80, wherein thetesting is performed in an open-loop type testing device.

EEE88. The post-processing device according to EEE80, wherein thetesting is performed in an closed-loop type testing device.

EEE89. The post-processing device according to EEE80, wherein thetesting is performed for a plurality of regions of the video.

EEE90. The post-processing device according to EEE89, wherein theplurality of regions comprise temporal regions of the video.

EEE91. The post-processing device according to EEE90, wherein theplurality of temporal regions comprise regions of changing shape andsize.

EEE92. The method according to EEE89, wherein the regions of the videoare determined by at least one of segmentation, object tracking, scenecut detection, edge detection, watershed, haursdorff method, K-means,motion detection, motion estimation, and motion analysis.

EEE93. The method according to EEE75, wherein the regions of the videoare identified using at least one of CGM, SVG, EPS, WMF, TIFF, BMP,JPEG, JPEG-2000, MPEG-2, and H.264, and MPEG4-AVC.

EEE94. A system, comprising:

-   -   an encoder configured to select at least one of a processing        technique, a category of processing techniques, and a series of        processing techniques to be utilized in decoder and encode the        selected techniques or an identifier of the selected        technique(s) in at least one of an encoded video stream and side        information related to the encoded video stream; and    -   a decoder configured to receive the encoded video stream and        decode the video stream using at least one of the selected        technique(s).

EEE95. The system according to EEE94, wherein the selected technique(s)are solely identified from the encoded video stream.

EEE96. The system according to EEE94, wherein the selected technique(s)are identified by the decoder from at least one of the related sideinformation and the encoded video stream.

EEE97. The system according to EEE94, wherein the techniques areselected based on at least one of quality, complexity, and cost.

EEE98. The system according to EEE94, wherein the decoder uses theselected processing technique to select another processing technique.

EEE99. The system according to EEE94, wherein the decoder maintains aninventory of processing techniques and selects a processing technique ortechnique(s) from the inventory most closely matching the technique(s)selected by the encoder.

EEE100. The system according to EEE98, wherein the decoder selectedtechnique is selected based at least one of complexity, cost, andquality.

EEE101. The system according to EEE98, wherein the decoder selectedtechnique is selected based at least in part on complexity in light ofan amount of processing power available in the decoder.

EEE102. A decoder configured to decode an encoded video steam using oneof a decoding process and a post-process identified in the video streamor via side information related to the video stream, wherein the decoderis configured to perform an additional analysis of at least one of thevideo stream and the decoded video stream utilized to provide an assistto the decoding process or post-process.

EEE103. The decoder according to EEE102, wherein the additional analysisutilizes information encoded in the encoded video stream or sideinformation.

EEE104. The decoder according to EEE102, wherein the additional analysisis directed at least in part by information encoded in the encoded videostream or side information.

EEE105. The decoder according to EEE102, wherein the additional analysiscomprises a refinement of information encoded in the video stream orprovided in side information.

EEE106. The decoder according to EEE105, wherein the refinementcomprises a lower cost.

EEE107. The post-processing device according to EEE75, furthercomprising a post-process analyzer configured to analyze at least one ofthe decoded video stream and the video stream prior to decoding, and,based on the analysis, direct at least part of at least one of adecoding process and the post processor.

EEE108. The post-processing device according to EEE107, wherein theanalyzer is directed based on information contained in the video streamor in side information related to the video stream.

EEE109. The method according to EEE40, further comprising the step ofpackaging a directive instruction for an additional analysis to beperformed by at least one of the decoder and a post-processor duringdecoding and/or post processing of the video data after decoding.

EEE110. A method comprising the steps of:

-   -   receiving a signal indicative of a processing technique to be        utilized on an encoded video data stream comprising downsampled        multiplexed views; and    -   performing at least one of a decoding and a post-processing of        the encoded video stream consistent with the received signal;        and    -   performing a further analysis of at one of the encoded video        stream and decoded/post-processed video stream and, based on the        analysis, performing at least one of improving the decoding,        improving the post-process, and performing a second        post-process.

EEE111. The method according to EEE110, further comprising any of thesteps described in EEEs 2-39.

EEE112. The method according to EEE110, wherein the step of performing afurther analysis comprises a refinement of the indicated processingtechnique.

EEE113. The method according to EEE112, wherein the method is performedby at least one of a media player, a set-top box, a converter box, and adisplay device.

The present invention may suitably comprise, consist of, or consistessentially of, any of element (the various parts or features of theinvention) and their equivalents as described herein. Further, thepresent invention illustratively disclosed herein may be practiced inthe absence of any element, whether or not specifically disclosedherein. Obviously, numerous modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as specifically describedherein.

What is claimed is:
 1. A non-transitory computer-readable medium storinginstructions for performing a decoding method, the decoding methodcomprising: receiving, at a decoder, a video bitstream comprising atleast one region comprising two images; receiving first metadata in thevideo bitstream identifying a first post-decoding process to be appliedto the at least one region; receiving second metadata in the videobitstream identifying second post-decoding process to be applied to theat least one region; selecting, by the decoder, one of the firstmetadata or the second metadata; and applying the one of the firstpost-decoding process or the second post-decoding process identified bythe selected metadata to the at least one region without applying theother of the first metadata or the second metadata to the at least oneregion, wherein the video bitstream further comprises a second regiondifferent from the at least one region, and wherein the one of the firstpost-decoding process or the second post-decoding process identified bythe selected metadata is not applied to the second region.
 2. Thenon-transitory computer-readable medium of claim 1, wherein the secondpost-decoding process is related to de-blocking.
 3. The non-transitorycomputer-readable medium of claim 1, wherein the second post-decodingprocess maps the at least one region to a target device.
 4. Thenon-transitory computer-readable medium of claim 1, wherein the secondpost-decoding process was selected during studio work prior to masteringthe video bitstream.
 5. The non-transitory computer-readable medium ofclaim 1, wherein the first post-decoding process is at least one ofde-interleaving and upsampling.